… stalled-job reaper

Two related fixes for the premature ``cleanup_async_job_resources`` path
that flips async_api jobs to FAILURE within ~30-55s of dispatch while
NATS is still delivering results:

1. ``Job.update_progress`` no longer silently bumps ``stage.progress = 1``
   when ``stage.status`` is in a final state. The bumped value made
   ``is_complete()`` return True and triggered cleanup mid-flight. The
   trigger was ``_update_job_progress`` writing ``status=FAILURE`` at
   partial progress as soon as ``failed/total`` crossed
   FAILURE_THRESHOLD (very easy early in a job — 1-2 errors out of the
   first few results). Progress is a measurement; leave it alone. The
   honest FAILURE trip happens when the stage actually reaches 100%.

2. Split ``Job.FAILED_CUTOFF_HOURS`` (originally added in PR #368 to
   hide failed jobs from UI listings after 3 days) into:
   - ``FAILED_JOBS_DISPLAY_MAX_HOURS = 24 * 3`` — original UI/API filter
   - ``STALLED_JOBS_MAX_MINUTES = 10`` — reaper threshold
   The 72h value was reused by PR #1227's stale-job reaper for an
   entirely different purpose, leaving stuck jobs invisible for 3 days.
   ``check_stale_jobs`` now defaults to 10 minutes against ``updated_at``
   (already in place from #1227), so a job whose worker pool stops
   pulling messages gets reaped within one Beat tick (~10-25 min).
   ``--hours`` arg on the management command becomes ``--minutes`` to
   match.

Together: fix #1 stops the false-positive FAILUREs; fix #2 ensures true
positives are caught quickly without depending on the misfiring coercion.

Co-Authored-By: Claude <noreply@anthropic.com>

…(3 days -> 10 min) (#1235)

* fix(jobs): stop coercing partial-progress to 100% on FAILURE, tighten stalled-job reaper

Two related fixes for the premature ``cleanup_async_job_resources`` path
that flips async_api jobs to FAILURE within ~30-55s of dispatch while
NATS is still delivering results:

1. ``Job.update_progress`` no longer silently bumps ``stage.progress = 1``
   when ``stage.status`` is in a final state. The bumped value made
   ``is_complete()`` return True and triggered cleanup mid-flight. The
   trigger was ``_update_job_progress`` writing ``status=FAILURE`` at
   partial progress as soon as ``failed/total`` crossed
   FAILURE_THRESHOLD (very easy early in a job — 1-2 errors out of the
   first few results). Progress is a measurement; leave it alone. The
   honest FAILURE trip happens when the stage actually reaches 100%.

2. Split ``Job.FAILED_CUTOFF_HOURS`` (originally added in PR #368 to
   hide failed jobs from UI listings after 3 days) into:
   - ``FAILED_JOBS_DISPLAY_MAX_HOURS = 24 * 3`` — original UI/API filter
   - ``STALLED_JOBS_MAX_MINUTES = 10`` — reaper threshold
   The 72h value was reused by PR #1227's stale-job reaper for an
   entirely different purpose, leaving stuck jobs invisible for 3 days.
   ``check_stale_jobs`` now defaults to 10 minutes against ``updated_at``
   (already in place from #1227), so a job whose worker pool stops
   pulling messages gets reaped within one Beat tick (~10-25 min).
   ``--hours`` arg on the management command becomes ``--minutes`` to
   match.

Together: fix #1 stops the false-positive FAILUREs; fix #2 ensures true
positives are caught quickly without depending on the misfiring coercion.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(jobs): log per-job diagnostic on stalled-job revoke

When the reaper revokes a job, log a single WARN line capturing the
state needed to triage "why was this stalled?" without grepping back
through prior tick logs:

  - minutes since last update vs threshold
  - previous_status, dispatch_mode, celery_state
  - per-stage progress + status summary
  - pointer to running_job_snapshots for prior NATS consumer state

Pairs with PR #1227's per-tick consumer snapshots: every 15 min, each
running async_api job gets a NATS state snapshot; this new line tells
operators which of those snapshots is the last one before revocation.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>